REPORT RESUMES 

ED 020 287 ud oos cs4 

THE STATE LEVEL PROBLEMS ASSOCIATED WITH ASSESSMENT OF THE 
IMPACT OF TITLE I. SYMPOSIUM. 

BY- MILLER. W.C. 

PUB DATE 67 

EDRS PRICE MF-$0.25 HC-$1 .04 24P. 

DESCRIPTORS- ^FEDERAL PROGRAMS. ^COMPENSATORY EDUCATION 
PROGRAMS. ^PROGRAM EVALUATION. ^DEVALUATION METHODS. ESTATE 
DEPARTMENTS OF EDUCATION. ADMINISTRATIVE PROBLEMS. FEDERAL 
STATE RELATIONSHIP. STATE SCHC-DL DISTRICT RELATIONSHIP. 

STUDENT NEEDS. SELECTION. EDUCATIONAL FIN NCE . PERSONNEL 
NEEDS. DATA COLLECTION. EDUCATIONAL OBJECTIVES. MEASUREMENT 
INSTRUMENTS. DISADVANTAGED YOUTH. ESEA TITLE 1 

OUTLINED IN THIS REPORT ARE STATE AND LOCAL PROBLEMS 
ASSOCIATED WITH ELEMENTARY AND SECONDARY EDUCATION ACT TITLE 
1 PROGRAM OBJECTIVES. SPECIALIZED PERSONNEL. DATA COLLECTION 
AND ANALYSIS. AND STABILITY OF ADMINISTRATION AND FUNDING. 
AMONG THE PROPOSALS FOR DEALING WITH THESE PROBLEMS ARE 
SUGGESTIONS FOR REGIONAL OFFICES TO ADMINISTER AND EVALUATE 
TITLE I PROGRAMS. AND FOR A NATIONAL REVIEW PANEL TO MAKE 
PROGRAM RECOMMENDATIONS TO CONGRESS. THE STANDARDIZATION OF 
DATA COLLECTION PROCEDURES AND THE DEVELOPMENT OF INFORMATION 
FEEDBACK ARE ALSO RECOMMENDED. CLB) 














fajjyfejki/fwfliriWiiirtol 



GO 



U.S. DEPARTMENT OF HEALTH, EDUCATION & WELFARE 
OFFICE OF EDUCATION 



THIS DOCUMENT HAS BEEN REPRODUCED EXACTLY AS RECEIVED FROM THE 
PERSON OR ORGANIZATION ORIGINATING IT. POINTS OF VIEW OR OPINIONS H 
STATED DO NOT NECESSARILY REPRESENT OFFICIAL OFFICE OF EDUCATION 
POSITION OR POLICY. 







O 

o 

\JJ 



.3 



SYMPOSIUM 

The State Level Problems Associated With 
Assessment of the Inq>act of Title 1 







m 



W. G. miler, Ph.D. 

Coordinator of Staff Consulting 

Computer Research Center 
University of South Florida 
Tampa, Florida 















Mi 






I . Introduction 

A. Relationships between State and Local or State and 
Federal Concerning Title I. 

B. Technical Competence for Assessments at the local school 
level. 

C. Assessment of Objectives - Is It Possible? 



II. The Problems 

A. Ambiguity of Objectives 

B. Direction of Projects 

C. Personnel Competencies 

D. Multiplicity of Measuring Instruments 

E. Non-Uniform Reporting Procedures 

F. Lack of Qualified Personnel and Equipment 

G. Need for Experimental Designs 

H. Prior Knowledge of Data to Collect 

I. Need for Feedback 

J. Criteria for Funding Under Title I 

K. Inclusion of Children into Multiple Projects 

L. How to Combat Educational Deprivation 

III. Some Proposals 

A. Regional Offices Responsible for Title I 

B. Determination of Priority on Needs of Educational 
Deprived Children 

C. Determination of Criterion for Including a Child in 
a Project 










■iiaaMSiaaiiMa^^^ ... 










D. Delimiting Tests Appropriate to Measure Objectives 

E. Standardization of Data Collection Procedures 



F. Development of Feedback 

G. National Review Panel 



IVe Summary 



I 



















Page 1 

Introduction: 

Not having had an opportunity to discuss these symposium 
topics with Dr. Froomkin, I must say beforehand that there no 
doubt exists some overlap of state problems with federal 
problems. Likewise there will probably be some topics covered 
herein that are common with Dr. Maxey*s and Dr. Foley's 
Insights. 

Let me also say that my encounter with Title I has been 
primarily as a coordinator of research to evaluate the impact 
of Title I for a given year in a given state ( Iowa- 1965-66) . 
Therefore ray comments are colored by an extremely limited 
scope of activity. Finally, it must be said that the agency 
for which I worked, the Iowa Educational Information Center, 
while supported by the Iowa Department of Public Instruction 
was also affiliated somewhat more strongly in practice with 
the Measurement Research Center and The University of Iowa. 
Therefore a certain autonomy of operation was felt and I was 
in the position of seeking data for research from the state 
department but not for the state department. This provided 
a certain advantage of being slightly removed from the 
battlefield itself yet close enough to make some observations. 

A. I would first like to malce some observations about 
relationships between local and state agencies and between 
state and federal agencies. Looking back to 1965, the 
timetables for implementation of Title I projects we.e 
quite brief. Evidently there was not sufficient time for 
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the Office of Education to design adequate forms and procedures 
to implement their objectives nor time to adequately orient 
state personnel to their thinking. Iirtiat can be said on this 
at the federal level is even more true at the state level. 

In state departments it was frequently necessary to shift 
personnel from one state department role to that of a role 
in the Title I area. Personnel thus placed were typically 
non~research oriented and over-taxed in their workloads. 
Consequently the articulation with local agencies left much 
to be desired ^ particularly in the area of evaluation of 
effectiveness. The sheer volume of papen^york involved in 
accounting for purchases made with Title I funds was itself 
impossible to adequately handle. The upshot was that local- 
state and state-federal relationships were strained regarding 
reporting procedures, deadlines for reports, dates for 
release of funds and guidelines for evaluation. 

These time-personnel problems have not disappeared with 
the ensueing years. Form revision always creates misunder- 
standings and additional work on the part of those completing 
them. IJhere one has established computer programs to produce 
analyses and reports, revisions in required data analyses 
and reports can be particularly expensive and time consuming. 
Other problems between local and state and between state and 
federal have also arisen .not as a consequence of timetables and 
report forms but for other reasons to be elaborated upon. 

B. Special consideration should be given to the problem 
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of professional competence of personnel available at state and 
local agencies. Many persons at the local school level 
responsible for administration of Title I projects have little 
or no experience In the designing of situations to assess 
Improvement In education , In collection of data^ and partlc~ 
ularly In analyzing data to make Inferences concerning the 



projects. It has been my experience and my associates 



experience to visit with local officials who feel adequate 
evaluation data consists of the local percentile ranks 



obtained by children on a single testing using an outdated 
instrument. In addition, I suspect It is not Impossible to 



find situations In which data are manipulated to prove a 
political point In order to utilize funds in a way desired 
by a few. Data thus reported are, at best, often misleading. 
At the State Department level, the task of assembling percen- 



tiles, stanlnes, standard scores and raw scores from many 
forms of numerous tests becomes a nightmare. The best that 



can be mustered up to evaluate some projects are statements 
like, "It seemed to turn out all right." Competent personnel 
at the local level might solve some of these problems. This 
solution is at best a remote possibility since It implies 



that many local school personnel would be trained in statistics 
and measurement a situation as likely to occur as the proverbial 



snowball In H. A more realistic solution will be offered later. 



It should be noted that state departments are staffed to a large 
degree by former public school principals and superintendents. 

1 shall not comment about their research and evaluative skills. 
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C. Now I would like to move toward the crux of what I feel 
to be state level problems. 

The first centers around the concept embodied in the 
Elementary and Secondary Education Act of 1965 itself. Section 
212 stated that *‘the commissioner shall submit to the Secretary 
of Healths Education and Welfare for transmission to the 
Congress on or before Deceinber 31, 1967, a full report of the 
operation of this title, including its effectiveness In improving 
the educational attainment of educationally deprived children. . . . 
Three substantial problems are thus introduced! 

(1) the definition of what constitutes "effectiveness'*, 

(2) the determination of improvement, and 

(3) the identification of educationally deprived children. 
Looking at the problem of defining "effectiveness," one may 

consider several alternatives: 

(a) Subjective feelings that children have benefited from 
experience encountered via Title I projects and 

(b) Statistically significant gains in pupil scores used 
to measure objectives of projects which would not 
have occurred without inclusion of pupils in the 
project. 

Most evaluators I suspect would prefer the latter alternative 
in that one learns to mistrust subjective judgements. Public 
school personnel are prone however to employ the first alternative. 
Reports from state department to the U.S. Office of Education 
contain considerable subjective material in evaluating 
"effectiveness." Guidelines eminating from the U.S.O.E. should 
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perhaps be more specific in defining "effectiveness." 

The second problem, that of determining "improvement" 
relates, of course, to the prior definition of "effectiveness." 

If one chooses the empirical route, a number of technical 
problems related to research design are presented. For example: 

(a) Are the stated objectives for a given project amenable 
to quantification and measurement? 

(b) Assuming (a) above to be true, do instruments exist 
for use? 

(c) Are the time spans involved for a given project adequate 
to detect measureable changes using available instruments? 

(d) Can the effect of the Title I project be "partialed 
out" from other environmental influences? 

(e) Are comparison groups (control groups) available for 
comparative studies? 

(f) Can data from different schools with similar project 
objectives be combined for analyses? 

The answer to the above questions is all too frequently "no." 
The satisfying of all the above conditions poses major problems 
for the local project director in determining what to do, hoi7 to 
do it, and when to do it. A procedure for meeting the above needs 
will be presented shortly. 

The third major problem stated above was the problem in 
identifying Title I children themselves. True enough, criteria 
concerning parental income have been forwarded. Obviously 
hox^ever, all lower income parents do not have educationally 
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deprived children. Guidelines fortunately have been broad 
enough to permit inclusion of other children; the total 
financial support being the major determinant of the numbers 
of children included in projects. VIhat seems to have occurred 
in a number of instances is that Title I funds have replaced 
or supplemented special education funds in many school board 
budgets. That is, funds from Title I are used to support 
existing programs for the educable mentally retarded and 
previous funds spent in this area are absorbed into the 
general budget. Thus frequently new programs were not 
generated. In these cases, as in others, I suspect the children 
Involved are frequently not those that are educationally retarded 
as a consequence of environmental stimulus restriction but 
reatrded as a consequence of biological or physiological con- 
straints. It would seem that the Intent of the Title I was to 
prov7 accelerated and individualized school experiences for 
child . who are educationally retarded as a consequence of 
their familial and cultural environment. Further clarification 
of criteria for inclusion in Title I projects would therefore 

be helpful. 

It should be noted that the structure of the E.S.E.A.. and 
Title I in particular is to attack special aspects or problems 
in education, not simply pump money into the general education 

budget . 

To summarize what I have been saying in this section let 
me rephrase what 1 feel are important issues pervading all of 
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Title I, at least at the state level, in the form of a cpmpound 
question: "What should be or are the objectives of Title 1 

projects, who do the projects affect and can the success in 
meeting the objectives be evaluated?" 
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II. I would now like to elaborate on specific problems encountered 
at the state level. Later I will offer some solutions to those 
problems . 

A. The first problem I see at the state level is the untenable 
objectives put forward by local agencies for their projects. Gross 
statements like "improve child’s self image" I find more than 
distasteful from our ability to assess the degree to which a 
"self-image" has been Improved. This is not to say such things 
don't exist or are unmeasureable. It simply lacks the specificity 
I feel should be embodied in any set of educational objectives. 

Classification of projects on the basis of objectives become 
a major problem for state department researchers desiring to 

summarize the types of projects funded. 

Some projects state an objective like "improve school attitude 
yet spend 90% of their allocation on a portable classroom or school 
room renovation for special education. There seem to exist in many 
projects a (pardon the expression) credibility gap between 
objectives and expenditures. 

B. A second major problem is the time span involved in 
assessing the degree to which objectives are attained. Nineteen 
sixty-five to sixty-six projects were late in getting established 
and were particularly prone to exist for a short duration. A 
serious question arises as to the sensitivity of even our best 
achievement instruments to assess the impact of a six week summer 
project. Assessment over longer periods of time are possible, 
of course, if the same objectives are maintained for sever^ years 
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and involve the same children and if data are carefully collected 
and maintained. This in itself produces Insurmountable problems 
for some schools, and in addition, yearly evaluations have been 
called for by state department and federal department officials. 

C. A third major problem (mentioned before) is the lack of 
skills of local and state department officials in collecting, 
analyzing and summarizing data in addition to maintaining longi- 
tudinal records. Many state departments simply are not able to 
attract personnel to do the job required or provide time and funds 
to do it . 



D. A fourth major problem is the diversity of instruments 
available and used for measuring educational objectives or assessing 
pupil characteristics. Even within specific grade levels and 
within a specialized area like reading skills, a wide variety of 
Instruments are available for use. Coupled with the multitude of 
norms available, different editions and forms of the same tests, 
and possible times for administration, the combining of data 
across various projects appears ludicrous. Further, there are 

also areas in which standardized instruments are not available. 

An example might be an instiniment to assess "health improvement." 

E. A fifth problem, related to the fourth, is tlis diversity 
between states and even within states, of the format for reporting 
project results. I have seen reports in which state report forms 
vere totally disregarded in favor of a locally contrived one. I 
suspect considerable variability existed among state reports 













Page 10 



although 1 can't attest to it. Perhaps a common reporting 
procedure is impossible for projects as potentially diverse as 
Title 1 permits. On the other hand, it should not be unreasonable 
to provide some reporting of results. Most disappointing is to 
receive a set of rai<r scores on a group of children with no refer- 
ences whatsoever as to what the scores represent. 

F. The sixth problem is the availability of personnel and 
equipment to analyze evaluative data. To analyze experimental 
and control group data administered before, during, and after a 
given Title I project duration and which is covaried with other 
data for statistical control presents a challenge, to say the 
least, for a principal with a desk calculator and a high school 
algebra background. For him it was a considerable challenge to 
even collect the data. Thus , it happened that ideal experimental 
data went uncollected because the possibility of analyzing the 
data was extremely remote. 

G. Another problem which presented itself to some schools 
which did have data collection and analysis capabilities was the 
experimental design problem itself. Many school personnel sincerely 
desired to compare special Title I efforts with traditional class- 
room approaches. One problem that immediately befell them was that 
if all the children who were educationally deprived were identified 
and placed in the Title I group, what children were left for a control 
group? Could covariance designs adequately evaluate special or 
experimental effects? And then there were problems of the testing 
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of several groups with repeated measures. Are available tests of 
achievement in English equally valid for children taught by a 
structural linguist and those taught by a grammarian? If not, must 
both groups receive tests designed for both methods of instruction 
to make comparison? And how does initial mental age affect growth 
under several instructional procedures? It can be seen that the 
testing program to assess the special effects of Title I involvement 
may itself be a project of scope large enough to warrant special 
funding! 

H. Reference was made previously to problems involved in 
changing forms for reporting data to the state department or to the 
U.S. Office of Education. Even more important than the implications 
for form changes is the problem of knowing befor_e the sta^ of a 
project , what data will be required in a report. Nothing is more 
frustrating to the local school than receiving a form 2/3 the way 
into a project which calls for data which should have been collected 
months before. (It*s almost as bad as receiving the allocation that 

far into it) . 

I. The next problem goes somewhat beyond Title I itself and 
involves many data collection activities of state departments. The 
problem, generally speaking, is to produce useabl e feedback to the 
local school. From the experiences of the Iowa Educational Infor- 
mation Center, I can tell you that it is a non-trlvlal problem. 
Local schools feel that they are in the position of providing data 
and more data - mch of it duplicated. The cry then is made that 
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all this is for naught - no one. ever sees what use is made of the 

■i 

data. Hence a serious need for feedback of data analyses is present. 

The critical point is that feedback must be of the nature that it 
can be used to actually modify behavior of teachers or school 

I 

administrators. 

i 

ij 

A personal mistake was made, for example, in thinking that if 

I 

schools received multiple copies of a miniature cumulative folder 
on each student, that teachers would desire and use such information. 

Many thousands of dollars later, the l.E.I.C. discovered that in 
many schools the information was never distributed and those that 

I 

did distribute it felt it came too late to be of use. The point to j 

i 

be made then is that the utility and design of feedback is a diffi- 
cult task - one that needs much development and testing. Feedback 
must Include detailed explanation of its importance and implications 
for actions to be taken and yet brief enough to be readily digested j 

by the busy schoolman. At least one important feedback document, j 

concerning data collected locally , statewide and nationally should 
be returned to each person directly involved with Title I projects. 



J, The next problem is one that concerns the state departments 



in their evaluation of requests for funds by local agencies. The 

s 

problem is differentiating between requests for funds to aid 
educationally deprived children or requests to meet general 
academic needs such as school nurses, guidance counselors, industrial 
education, etc. typically required for receipt of state legislative 
appropriations. I find it difficult to legitimately employ Title I 
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funds to meet these standards. These are mlnlmums that are to be 
met irrespective of E.S.E.A. funds. Yet a number of projects of 
my recollection involved funds to help meet these minimum standards. 
There are, of course, excellent arguments in favor of supporting 
school budgets to meet such general school needs but they do not 
seem to be within the spirit of Title I. I feel therefore, that one 
problem deserving of attention at the state level, is differentiating 
more strictly, the general nature of the use to which Title I funds 
are employed. Other financial needs should be met through other means. 



K. Perhaps this next point is not a problem but an observation. | 

Since a number of federally funded projects may exist within a local j 

i 

school, it is of some interest to know what the degree of overlap is ] 

from project to project with respect to the students affected. It I 

was our experience in studying Title I involved children, that a I 

given child may be receiving services throu^ as many as four or 

■j 

five separately funded projects. A weighting factor of the dollars 

spent per child, when evaluating academic improvement, might reveal 

an important summary concerning the overall effectiveness of Title I 

as it relates to various project objectives. | 



L. Finally, let me ask one naive question. What information 
do we have concerning the relationship of economic deprivation to 
educational growth? I am not personally convinced that it is the 
economic deprivation of the schools and their programs that is the 
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most relevant variable in educational growth. It would appear that 
the economic deprivation of the family, and the resultant attitudes 
and behaviors associated with the economic standing of the family 
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which might be an even more important factor in educational 
attainment. It this be true, wouldn't efforts at improving the 
socio-economic characteristics of the family be ft more promising 
avenue to escplore in order to attain educational improvement of 
youth? Ascertaining the validity of this assertion conceivably 
could vastly affect the nature of projects funded under E.S.E.A. , 
and in fact, the very continuance of federal support of the 
Elementary and Secondary Education Act. 
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III. Noi/ that 1 have put forth what 1 consider to be some problems 
concerning Title I at the state level, let me propose some changes 
designed to cope with these problems. 

A. Let me consider first the organizational structure for 
administration and evaluation of Title I activities. I feel there 
would be considerable advantage in the vesting of certain admini- 
strative responsibilities into Interstate or regional centers. 
Specifically I would propose the following: 

(a) Creation of six regional administrative offices of the 

U.S. Office of Education - a South Eastern, a South Central, 
a South Western, a North Western, a North Central and a 
North Eastern regional office. 

(b) Each region would determine the -procedures, forms and 
timetables for data collection, analyses and dissemination 
of data. 

(c) Regional offices would process applications and disburse- 
ments of f-imds for Title I projects. 

(d) Data processing would be established within these regions 
to properly and uniformly collect, analyze and disseminate 
data. 

(e) Regional offices would provide consultants to aid in 
determining appropriate statistical designs, measuring 
Instruments and project timetables for funded projects. 

(f) Regional offices would maintain data banks of information 
collected on pupils involved in Title I projects and 
produce yearly longitudinal studies and summaries of 
these data. 
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This proposal is made as a device to meet the problems of 
reducing variations among states in their practice of administering 
Title I and to relieve the state department from an impossible data 

# 

collection and analysis responsibility and to increase the efficiency 
of administration in general. State departments would maintain the 
responsibilities of providing the regional offices with data concerning 
per capita income. Intrastate educational and socio-economic 
deficiencies, general educational standards and financial support 
to local schools, and indexes of general pupil achievement throughout 

the state. 

B. I would propose that the U.S. Office of Education, along 
with specialists in the above proposed regions establish a hierarchy 
of needs amenable to attacks through Title I projects to serve as 
guidelines for establishment of local projects. These would not 

be meant to restrict local evaluation of needs but serve solely to 
establish the intents of the Congress concerning Title I. 

C. I would propose that guidelines be developed cooperatively 
by the proposed regions for establishing criteria for Including 
children into Title I projects. These guidelines might take the 
form of sample criteria for projects designed to meet the needs 

outlined in (B) above. 

D. I would propose the selection of a repertoire of nationally 
normed tests to evaluate Title I objectives. Thus a selection of 
three or four reading tests, arithmetic tests, general achievement 
test batteries, etc. x^ould serve as a basis for natiom^ide 
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comparative data. Where appropriate instruments are unavailable, 
regional, offices would be charged with the responsibility for 
development and testing of them. 

E. I propose the standardization of data collection instruments 
and procedures within each regional office to collect data necessary 
for evaluation and longitudinal growth studies. (See appendix) 

F. I would propose each regional office would be charged with 
producing three major reports; 

(a) A report to the U.S. Office of Education concerning the 
progress and effectiveness of Title I activities. 

(b) A report to states within each region showing comparative 
data for regions, states and communities. 

(c) A report to each contributing local school agency 
summarizing the analysis of data from that agency and 
providing detailed interpretations and suggestions for 
implementing changes. 

G. I propose the directors of the regional centers for Title 
I plus 6 other nationally recognized persons from the U.S. Office 
of Education, college and universities and private industry form a 
National Review Panel on Title I to make recommendations to the 
Congress concerning this section of E.S.E.A. 
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IV. Summary 

In summary » it may be said that local and state personnel 
involved with Title I have had problems concerning objectives, 
specialized personnel, data collection and analysis, and stability 
of administration and funding. It was proposed that regional 
centers be established to administer Title I, collect and analyze 
data and to classify and evaluate Title I objectives and evaluate 
the effectiveness in meeting these objectives. The structure for 
these regional offices already exists of course in various funded 
centers and in present regional offices of the U.S. Office of 



Education. 
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